pemprosesan bahasa tabii/ buku rujukan natural language 1 pengenalan.pdf · zsistem soal- jawab ......

1

TP2663 Asas Pemprosesan Bahasa Tabii 1

Pemprosesan Bahasa Tabii/Natural Language Processing (NLP)

Pengenalan


Pemprosesan Bahasa Tabie (NLP)

Buku Rujukan

James Allen, Natural Language Understanding, 1995. The Benjamin/cummingsPublishing.

Mana-mana bukuberkaitan natural language dan prolog programming

PeperiksaanAkhir70 %

Tutorial /Kuiz30 %

Kedatangan? %


LatarbelakangApakah Pemprosesan Bahasa Tabie

NLP merupakan kaedah bagaimana untuk menggunakankomputer untuk menganalisa dan memahami bahasa manusia(Cynthia 2000)

Proses untuk membangunkan model pengkomputeran bagimemahami bahasa tabie

INPUT : teks bahasa tabieOUTPUT : perwakilan maksud bagi suatu ayat


Applications for NLP

Database query interfaceMachine TranslationQuestion Answering SystemsTutoring SystemsFact ExtractionInformation Retrieval

Text retrieval, categorization, routing, filteringSpeech / Spoken Language Understanding

2


NLP applications: ExamplesContoh penggunaan NLP

answering the phone, and replying to a question understanding the text on a Web page to decide who it might be of interest to translating a daily newspaper from Japanese to English (an attempt is made to do this already) understanding text in journals / books and building an expert systems based on that understanding

Contoh aplikasi komersialLanguage tools oleh Google: http://www.google.com/language_tools?hl=enSearch engine AskJeeves: http://www.ask.com/Terjemahan Arab-English on-line:

http://tarjim.ajeeb.com/ajeeb/default.asp?lang=1Eliza – antara yang pertama dibangunkan dan banyak versi lain spt Frank, Meliza dll.


Aplikasi NLP

Boleh dikelaskan kepada 2:Aplikasi berasaskan teksAplikasi berasaskan dialog


Aplikasi NLP Aplikasi berasaskan teks

Melibatkan pemprosesan menulis teksMendapatkan dokumen yang sesuai berdasarkan sesuatutopikMendapatkan maklumat dari mesej atau artikelberdasarkan sesuatu topikMenterjemah dokumen dari satu bahasa ke bahasa yang lainMembuat ringkasan sesuatu teks untuk tujuan tertentu

(teknik biasa : kata kunci; Menangani kueri yang kompleks : dapatkan semua artikel berkaitan hargasaham yang jatuh pada tahun 1980an dan naiksemula pada tahun 2003)


Aplikasi NLPAplikasi berasaskan dialog

Melibatkan komunikasi antara manusia dan mesinSistem soal- jawabPerkhidmatan pelanggan automatik melalui telefonSistem tutoran

3


Kinds of knowledge needed?

Consider the following interaction with HAL the computer from 2001: A Space Odyssey

Dave: Open the pod bay doors, Hal.HAL: I’m sorry Dave, I’m afraid I can’t do that.


Knowledge needed to build HAL?

Speech recognition and synthesisDictionaries (how words are pronounced)Phonetics (how to recognize/produce each sound of English)

Natural language understandingKnowledge of the English words involved

What they meanHow they combine (what is a `pod bay door’?)

Knowledge of syntactic structureI’m I do, Sorry that afraid Dave I’m can’t


What’s needed?Dialog and pragmatic knowledge

“open the door” is a REQUEST (as opposed to a STATEMENT or information-question)It is polite to respond, even if you’re planning to kill someone.It is polite to pretend to want to be cooperative (I’m afraid, I can’t…)What is `that’ in `I can’t do that’?

Even a system to book airline flights needs many of this kind of knowledge


Question AnsweringWhat does “door” mean?What year was Abraham Lincoln born?How many states were in the United States when Lincoln was born?Was there a military draft during the Hoover administration?What do US scientists think about whether human cloning should be legal?

4


The History of NLP

The Dartmouth Conference (1956)The birth of artificial intelligence (AI)

Heuristic search (Newell & Simon, 1956)LISP (McCarthy, 1960)Machine Translation


Early Research in Machine Translation

I must go home.Saya mesti pergi kediaman.

Hydraulic ramWater sheep

Out of sight, out of mindBlind, crazy

The spirit is willing but the flesh is weak.The vodka is good but the meat is rotten.


Conclusion: MT is ImpossibleA mathematician named Bar-Hillel claimed that machine translation could not be accomplished by mapping one set of strings to another. Example:

The pen is in the boxThe box is in the pen

The correct translations requires knowledge about:The typical sizes of boxes and writing instruments.The concept of “in”.Spatial reasoning.


Ambiguity

Find at least 5 meanings of this sentence:I made her duck

5


AmbiguityFind at least 5 meanings of this sentence:

I made her duckI cooked waterfowl for her benefit (to eat)I cooked waterfowl belonging to herI created the (plaster?) duck she ownsI caused her to quickly lower her head or bodyI waved my magic wand and turned her into undifferentiated waterfowlAt least one other meaning that’s inappropriate for gentle company.


Ambiguity is PervasiveI caused her to quickly lower her head or body

Grammar: “duck” can be a noun (waterfowl) or a verb (move body)

I cooked waterfowl belonging to her.Grammar: “her” can be a possessive (“of her”) or dative (“for her”) pronoun

I made the (plaster) duck statue she ownsMeaning: “make” can mean “create” or “cook”


Ambiguity is PervasiveGrammar: Make can be:

Transitive: (verb has a noun direct object)I cooked [waterfowl belonging to her]

Ditransitive: (verb has 2 noun objects)I made [her] (into) [undifferentiated waterfowl]

Action-transitive (verb has a direct object and another verb) I caused [her] [to move her body]


Ambiguity is PervasivePhonetics!

I mate or duckI’m eight or duckEye maid; her duckAye mate, her duckI maid her duckI’m aid her duckI mate her duckI’m ate her duckI’m ate or duckI mate or duck

6


Jenis pengetahuan berkaitanNLP

pengetahuan umum mengenai struktur bahasapengguna di duniaPengetahuan dunia

bagaiman sesuatu ayat itu diguna pada keadaan yang berbezaPragmatik

hubungan antara ayat : konteks. Bagaimana ayatdipermulaan memberi kesan secara langsung kepadapenerangan ayat berikutnya

Discourse

maksud sesuatu perkataan dan bagaimana maksud inidicantumkan dalam suatu ayat membentuk ayat yang bermaksud

Semantik

mengambil kira bagaimana perkataan dihubungkan bagimembentuk ayat yang betulSintaks

bagaimana sesuatu perkataan itu dibina (prefiks dansuffiks)Morphology

mengambil kira bagaimana sesuatu perkataan itudibunyikan

Phonetik danphonologikal


Contoh

Pertimbangkan ayat berikut:1. Languages is one of the fundamental aspects of

human behaviour2. Green frogs have large noses.3. Green ideas have large noses.4. Large have green ideas nose.

Jelaskan perbezaan di anatara sintaks, semantik dan pragamatik…..


Word sense

Perkataan yang mempunyai pelbagaimaksud – dipanggil senses.Contoh: perkataan cook -> boleh mewakliverb atau noun.Faktor ambiguity (kesamaran) inimenyebabkan sistem sukar untuk membuatinference yang diperlukan untuk memodelkankefahaman sesuatu ayat/perkataan.


Konsep Asas Linguistik

Morphology & PhonologySyntaxSemantikPragmatik

7


Morphology & PhonologyMorphology merupakan pengajian bagaimanaperkataan boleh dipecahkan kepada bahagian-bahagianyang bermaksud.

contoh : penyempurnaan ( peny sempurna an )

Perkataan juga dirujuk sebagai item leksikal. Dan Leksikonmerupakan struktur data yg digunakan untuk menjejak maklumatyang diperlukan untuk memproses perkataan dalam ayat.Teori bahasa juga menekan ciri-ciri perkataan seperti singular, plural, transitive, intransitive, first or third person (kombinasi ayat)

Phonology : mengkaji unit bunyi yang mungkin di dapati danbagaimana untuk mencantum diantara satu dengan yang lain

Prefiks kata akar suffiks


Sintaks/Sintaksis (Syntax)Mengkaji struktur yang sah bagi suatu bahasa.Terbahagi kepada beberapa kategori

Noun, verb dan adjectiveSintaks pengetahuan merupakan peraturan yang yangmenggabungkan kategori di atas kepada frasa dan menjadi ayat. S NP VP NP Det N VP V NPPeraturan menggabungkan frasa kepada struktur yang lebih besar : grammar.Jenis grammar: context-free atau struktur frasa. Grammar mengandungi

Set simbol terminal Set simbol non-terminalSimbol mulaSet menulis semula peraturan

TP2663 Asas Pemprosesan Bahasa Tabii 27 TP2663 Asas Pemprosesan Bahasa Tabii 28

Lexical processingLook up the individual words in a dictionary. It may not be possible to choose a single correct meaning, since there may be more than one. The process of determining the correct meaning of individual words is called word sense disambiguation or lexical disambiguation. For example, "I'll meet you at the diamond" can be understood since atrequires either a time or a location. This usually leads to preference semantics when it is not clear which definition we should prefer.

Sentence-level processingThere are several approaches to sentence-level processing. These include semantic grammars, case grammars, and conceptual dependencies.

Example: Semantic processing determines the differences between such sentences as "The cat is in the pen" and "The ink is in the pen."

8


Sintkas: kesamaran strukturPlease close the windows

Please close the windows

Please close the windows



The boy saw the man on the hill with the telescope


Sintaks: kesamaran struktur

Pertimbangkan ayat:

Time flies like an arrow.

Metaphor: time passes quicklyDeclarative: insects have an affinity for arrowImperative: measure the time of the insects

9


Syntax

Syntax tidak memberikan sebarang makna…..

Colourless green ideas sleep furiouslyFire match arson hotelDog collar vs flea collarPlastic can food can cover


Semantik

Mengkaji maksud bagi unit bahasa sama adaianya perkataan, ayat atau keseluruhandiscourse.

Mengkaji maksud satu-satu perkataan danbagaimana maksud perkataan digabung kepadamaksud ayat. contoh

(1) I took the blue book(2) the blue book spoke about its small feet.

Secara sintak kedua-duanyabetul tetapi dari segi

semantik ayat 2 salah


Semantik: kesamaran leksikal

I walk to the bank …of the river

to get money

The bug in the room …was probably planted by spiesflew out the windows


Semantik: kesamaran leksikal

10


Discourse: coreference

Hubungan di antara ayat; konteks:President JFK was assassinated.The president was shot yesterday.Relatives said that John was a good father.JFK was the youngest president in history.His family will bury him tomorrow


PragmatikMengkaji appropriateness, necessity and sufficiency in language use.Ia juga mengambilkira bagaimana ayat berkaitantara satu dengan yang lain.Pengetahuan pragmatik mengandungi maklumatmengenai ayat yang mungkin mangandungi maksudkonteks semasa.Adakalanya dirujuk sebagai “rules of cenversation”

Can you tell me what time it is?Could I have the salt?


Discourse and Pragmatic Processing

To understand most sentences, it is necessary to know the discourse and pragmatic context in which it was uttered. In general, for a program to participate intelligently in a dialog, it must be able to represent its own beliefs about the world, as well as the beliefs of others (and their beliefs about its beliefs, and so on). The context of goals and plans can be used to aid understanding. Plan recognition has served as the basis for many understanding programs - - PAM is an early example. Speech acts can be axiomatized just as other operators in written language, except that they require modal operators to describe states of belief, knowledge, et al.


World knowledgeGeneral knowledge about the world, common sense.

Contoh: Language untuk memulakan, menamatkan ataumengakhiri perbualan.

Ali went for dinner at a ‘warong’ stall. He ordered a fried mee. He paid and went

home.

What did Ali eat for dinner?Who cooked the fried mee?

Where did he paid?Did he receive a bill?

11


Organisasi Sistem NLP

Rujuk Rajah 1.5


TP2663 Asas Pemprosesan Bahasa Tabii 43 TP2663 Asas Pemprosesan Bahasa Tabii 44

Latihan1. Camkan dan nyatakan aspeks kesamaran (sintaktik

atau semantik) yang wujud dalam ayat berikut.a) If you saw a heat wave, would you wave back.b) We will sell gasoline to anyone in a glass container.c) 38 years on the spot.d) Some pieces of P.Ramlee sold at Art Actione) Juvenile court to Try Shooting Defendant.f) Tiga tahun di universiti masih di takuk lama.g) Bentangkan di hadapan pensyarah.h) Apabila melihat buaya tembaga, Ahmad lari lintang

pukang.i) Program pengeditan video itu sangat sesuai.

12


End of Topic 1

Dr. Shahrul Azman Mohd. NoahJabatan Sains Maklumat, FTSMUniversiti Kebangsaan Malaysia


pemprosesan bahasa tabii/ buku rujukan natural language 1 pengenalan.pdf · zsistem soal- jawab ......

Documents