Puma::Token Class Reference

#include <Puma/Token.h>

Inheritance diagram for Puma::Token:

List of all members.

---

Detailed Description

Abstraction of a language token.

A token is created by a scanner (see Puma::CScanner) as part a token chain (Puma::Unit). It encapsulates a lexical unit of the scanned text.

The information provided about a lexical unit is

• the token type
• the location in the scanned text
• the portion of the scanned text represented by this token
• the language of the token

Public Types
enum	{   ID_END_OF_FILE = -1,   ID_UNKNOWN = -2,   ID_ERROR = -3,   ID_WARNING = -4 }
	Special token types. More...
Public Member Functions
	Token (int type=0, LanguageID lang=LanguageID(0), const char *text="")
	Constructor.
	Token (const Token &copy)
	Copy-constructor.
virtual	~Token ()
	Destructor.
void	print (ostream &os) const
	Print the token text on the given stream.
virtual ListElement *	duplicate ()
	Duplicate this token.
Unit *	unit () const
	Get the unit this token belongs to.
void	reset (int type=0, const char *text=0, LanguageID lang=LanguageID(0))
	Reset the token object.
void	location (const Location &loc)
	Set the location of the token (usually the line and column in a file).
void	cont_lines (Array< int > *cl)
	Set the continues lines appeared in this token.
int	line_breaks () const
	Count the number of line breaks in the text of this token.
void	macro_generated ()
	Set the token as being macro generated.
void	macro_call ()
	Set the token as being a macro call.
const char *	text () const
	Get the token text.
DString &	dtext () const
	Get the token text.
char *	get_static_text () const
	Get the static token text.
int	type () const
	Get the type of the token (see Puma::CTokens).
const Location &	location () const
	Get the location of the token (usually the line and column in a file).
bool	is_macro_generated () const
	Check if the token is macro generated.
bool	is_macro_call () const
	Check if the token is a macro call.
bool	is_core () const
	Check if this is a core language token.
bool	is_identifier () const
	Check if the token is an identifier.
bool	is_keyword () const
	Check if the token is a keyword.
bool	is_wildcard () const
	Check if this is a wildcard token.
bool	is_preprocessor () const
	Check if this is a preprocessor token.
bool	is_directive () const
	Check if this is a compiler directive.
bool	is_whitespace () const
	Check if this is a white-space token.
bool	is_comment () const
	Check if this is a comment.
bool	is_macro_op () const
	Check if this is function-like macro operand.
bool	is_open () const
	Check if this is a left parenthesis.
bool	is_comma () const
	Check if this is a comma.
bool	is_close () const
	Check if this is a right parenthesis.
void *	operator new (size_t)
	Own operator new reusing memory.
void	operator delete (void *)
	Own delete operator.
Static Public Attributes
static LanguageID	comment_id
	Comment tokens.
static LanguageID	pre_id
	Preprocessor tokens.
static LanguageID	dir_id
	Compiler directives.
static LanguageID	cpp_id
	C/C++ core language tokens.
static LanguageID	white_id
	White-spaces.
static LanguageID	wildcard_id
	Wildcards.
static LanguageID	keyword_id
	Keywords.
static LanguageID	open_id
	Left parenthesis.
static LanguageID	macro_op_id
	Function-like macro operands.
static LanguageID	identifier_id
	Identifiers.
static LanguageID	comma_id
	Comma.
static LanguageID	close_id
	Right parenthesis.

---

Member Enumeration Documentation

anonymous enum

Special token types.

Enumerator:

ID_END_OF_FILE	End-of-file token.
ID_UNKNOWN	Unknown token type.
ID_ERROR	Error token type.
ID_WARNING	Warning token type.

---

Constructor & Destructor Documentation

Puma::Token::Token	(	int	type = 0,
		LanguageID	lang = LanguageID(0),
		const char *	text = ""
	)

Constructor.

Parameters:

	type	The token type.
	lang	The language of the token.
	text	The optional token text (defaults to the empty string).

Puma::Token::Token

(

const Token &

copy

)

Copy-constructor.

Parameters:

copy

The token to copy.

virtual Puma::Token::~Token

(

)

[virtual]

Destructor.

Destroys the token text.

---

Member Function Documentation

void Puma::Token::print

(

ostream &

os

)

const

Print the token text on the given stream.

Parameters:

os

The output stream.

virtual ListElement* Puma::Token::duplicate

(

)

[inline, virtual]

Duplicate this token.

Returns:

The copy of this token. Has to be destroyed by the caller.

Implements Puma::ListElement.

Unit* Puma::Token::unit

(

)

const [inline]

Get the unit this token belongs to.

void Puma::Token::reset	(	int	type = 0,
		const char *	text = 0,
		LanguageID	lang = LanguageID(0)
	)

Reset the token object.

Parameters:

	type	The new token type.
	text	The new token text.
	lang	The new token language.

void Puma::Token::location

(

const Location &

loc

)

[inline]

Set the location of the token (usually the line and column in a file).

Parameters:

loc

The location.

void Puma::Token::cont_lines

(

Array< int > *

cl

)

[inline]

Set the continues lines appeared in this token.

Parameters:

cl

The continues line array.

int Puma::Token::line_breaks

(

)

const

Count the number of line breaks in the text of this token.

void Puma::Token::macro_generated

(

)

[inline]

Set the token as being macro generated.

void Puma::Token::macro_call

(

)

[inline]

Set the token as being a macro call.

const char* Puma::Token::text

(

)

const

Get the token text.

DString& Puma::Token::dtext

(

)

const

Get the token text.

char* Puma::Token::get_static_text

(

)

const

Get the static token text.

Only for keywords, operators, and so on with a constant token text.

Returns:

The static text or NULL for tokens with dynamic text (like identifiers).

int Puma::Token::type

(

)

const [inline]

Get the type of the token (see Puma::CTokens).

const Location& Puma::Token::location

(

)

const [inline]

Get the location of the token (usually the line and column in a file).

bool Puma::Token::is_macro_generated

(

)

const [inline]

Check if the token is macro generated.

bool Puma::Token::is_macro_call

(

)

const [inline]

Check if the token is a macro call.

bool Puma::Token::is_core

(

)

const [inline]

Check if this is a core language token.

bool Puma::Token::is_identifier

(

)

const [inline]

Check if the token is an identifier.

bool Puma::Token::is_keyword

(

)

const [inline]

Check if the token is a keyword.

bool Puma::Token::is_wildcard

(

)

const [inline]

Check if this is a wildcard token.

bool Puma::Token::is_preprocessor

(

)

const [inline]

Check if this is a preprocessor token.

bool Puma::Token::is_directive

(

)

const [inline]

Check if this is a compiler directive.

bool Puma::Token::is_whitespace

(

)

const [inline]

Check if this is a white-space token.

bool Puma::Token::is_comment

(

)

const [inline]

Check if this is a comment.

bool Puma::Token::is_macro_op

(

)

const [inline]

Check if this is function-like macro operand.

bool Puma::Token::is_open

(

)

const [inline]

Check if this is a left parenthesis.

bool Puma::Token::is_comma

(

)

const [inline]

Check if this is a comma.

bool Puma::Token::is_close

(

)

const [inline]

Check if this is a right parenthesis.

void* Puma::Token::operator new

(

size_t

)

Own operator new reusing memory.

void Puma::Token::operator delete

(

void *

)

Own delete operator.

---

Member Data Documentation

LanguageID Puma::Token::comment_id [static]

Comment tokens.

LanguageID Puma::Token::pre_id [static]

Preprocessor tokens.

LanguageID Puma::Token::dir_id [static]

Compiler directives.

LanguageID Puma::Token::cpp_id [static]

C/C++ core language tokens.

LanguageID Puma::Token::white_id [static]

White-spaces.

LanguageID Puma::Token::wildcard_id [static]

Wildcards.

LanguageID Puma::Token::keyword_id [static]

Keywords.

LanguageID Puma::Token::open_id [static]

Left parenthesis.

LanguageID Puma::Token::macro_op_id [static]

Function-like macro operands.

LanguageID Puma::Token::identifier_id [static]

Identifiers.

LanguageID Puma::Token::comma_id [static]

Comma.

LanguageID Puma::Token::close_id [static]

Right parenthesis.